mastodon.matrix.org is part of the decentralized social network powered by Mastodon.

Server stats:

8
active users

Learn more

Element

🚨Nation-scale Matrix deployments will fail if built on the community version of Synapse.

The community version of Synapse is not designed or intended for use by commercial Matrix hosting providers to serve huge nation-scale deployments. Just as a suspension bridge has a weight limit and will collapse if you exceed it - the same goes for community Synapse.

Deployments supporting millions of users need Synapse Pro.

element.io/blog/scaling-to-mil

Element Blog · Scaling to millions of users requires Synapse ProNation-scale Matrix deployments will fail if built on the community version of Synapse. Huge deployments need a different architecture, which is what Synapse Pro delivers.

@element As a tax payer I want my government to use open source. Since Synapse Pro isn't, it's simply not usable for such cases.

Thank you for detailing the need for doing the same development to the open source Synapse as you have already done on Synapse Pro. I hope you do agree, it would be a pity to have to fork.

@troed We'd also like everything to be open source. But the reality is that even with AGPL, we're still stuck in the pattern that enormous deployments use FOSS Synapse without contributing to its dev/maintenance costs.

Every time an opportunity that we're counting on to fund the team turns around and says "oh, FOSS Synapse is good enough, and we can use it for free, bye" we find ourselves needing to provide a very concrete reason not to freeride - hence Synapse Pro.

@element Do you know of any successful open source businesses that have used this particular method?

"How to make money from FOSS" is an age old problem - but "the free version has a sucky implementation" is, afaik, not the most successful one.

@troed we're not remotely trying to do "the free version has a sucky implementation". FOSS Synapse will keep improving, including perf (but not scalability) work. The model is more like MySQL Enterprise (back int he day). "If you want a massive cluster with a guaranteed SLA, here's our paid clustering product".

@element Right, so let me make a quick rewrite:

"Scaling is hard!

We're seeing many successful Matrix deployments using Synapse for which we are extremely proud and happy. We're also seeing some running into scaling problems as they approach hundreds of thousands of users.

Element is here to help. We're the biggest contributor to the open source Synapse community, and as such we have unparalleled expertise when it comes to optimizing very large deployments. With our performance libraries, that are specifically crafted for Synapse deployments that can reach millions of users, we can get the performance of massive - and costly - server solutions down to something that's much more manageable.

If this is a good fit for your deployment plans, please reach out!"

@troed that is a great pitch - and is fairly close to the original Synapse Pro announcement from element.io/blog/synapse-pro-sl. The only problem is that then a bunch of big deployments said "that's nice, we'll risk it anyway". Hence the harsher "seriously, you will fail if you try to wing it" wording here.

Element Blog · Synapse Pro slashes costs for running nation-scale Matrix deploymentsSynapse Pro is Element’s best practice Matrix homeserver. It transforms the performance and economics of huge public sector deployments.

@element Ok so now I'll need to preface the next part: I am a former Head of SaaS that built a commercial product on top of a FOSS stack. We took the decision, multiple times, which parts to use as-is, which to pay for commercial support and what that cost/benefit looked like.

I'm sorry, you cannot scare potential customers into choosing your Pro product* over the FOSS offering. If they have the budget and their mission is clear, they will pay. If they don't, they'll need to miss their internal targets _before_ they're able to acquire that budget.

The only thing you can do is to wish such a customer good luck, and then sneakily have your sales personnel check in every now and then. If your Pro product is cheaper for their use case, they will come back and pay you once they've figured out they weren't as smart as they hoped.

Now, I asterisked "your Pro product" - because the big difference between your original product announcement and my rewrite is how you phrase Synapse Pro as _a different product_ and not just as paid consultancy/contract to optimize a Synapse deployment. That way you're basically telling your customer they have a one time choice between Pro and Community, and if they choose Community there's no easy "come help us!" path forwards once they run into scaling issues.

I do understand from this communication that things are tight though, and I truly hope that the EU, or individual governments, can finally understand that this is the area in which they should spend money. But, back the original post. As a dev, and former head of blabla, that post really came out as "we made the FOSS version sucky and put the good code into our proprietary product".

fwiw

@troed yup your point makes sense - especially as Synapse Pro very much *is* designed to be dropped into speed up FOSS Synapse when needed. Well, we can always do a 3rd blog post with spin to see how it lands... :)

@element @troed

Sell that product to private companies then, not to public institutions.

@element Would you be able to name a price if one such "nation-scale deployment" wanted to pay you to switch Synapse Pro to AGPL and maintain it as such for, say, 5 years?

@nemobis the funding gap we're try to plug for Synapse and associated Matrix dev is very roughly around $5M/y. So over 5 years, that'd be $25M.

@element Thanks! A very reasonable price if you ask me. Italy is planning to spend *billions* to supposedly secure their insecure military messaging at the network layer.

@nemobis yup, and be vendor-locked to SpaceX in doing so: bloomberg.com/news/articles/20.

Another solution here could be if governments decided to actually support FOSS by funding its maintenance & dev by subscribing to the paid product, rather than freeriding. ZenDiS in Germany is a great example of this. But unfortunately, they're almost unique in this, hence Synapse Pro.

Bloomberg · Italy Plans $1.6 Billion SpaceX Telecom Security Services DealBy Daniele Lepido

@element @nemobis oi, you cant do that. Either you complain about vendor lock in, or you can create vendor lock in yourself. You clearly chose to be the type of company that wants their customers locked to their product, so you really can't now complain that your competitors do the same.

@element @troed oh yes, very sad there is not a single working, widely deployed open source project used by nations all over the world.

@teto @troed tl;dr is more "need to be able to keep paying people to work on this".

@element @troed donations seem to work for KDE just fine

@teto @troed KDE get 350K EUR/y according to their annual report, which is barely enough to pay for the matrix.org hardware costs, let alone anyone to work on Matrix, unfortunately.

@element @troed you could've done the obvious and sell matrix as a service or sell support for enterprise customers like Canonical. A "pro" version will just make the project look laughable to the open source community. Simple as.
@teto @troed @element Canonical is actually selling "Pro" edition of Ubuntu
@kdecherf @troed @element but not because it's "faster". Any normal ubuntu version works just fine. And it's still open source. They just give you faster security patches or smth. Most admins check for that stuff anyways. At no company I worked at was ubuntu pro ever used.
@teto @troed @element "here's our paid software but i guess you can use the crummy old version with scalability problems for free" is not a good argument for why people should use your software
everyone who sees this knows what that means will happen over the coming months and years
>sorry that feature is no longer available in the free version, would you like to buy the pro version?
>haha whoops we've intentionally crippled this functionality of the free version, would you like to buy the pro version?
>you're only allowed to have 10 users on the free version now, would you like to buy the pro version?
>we've deliberately broken compatibility with this in the free version, would you like to buy the pro version?
>the free version now has ads in all channels, would you like to buy the pro version?

offering enterprise support + managed cloud hosting is the industry norm, not this
@skylar @teto @troed @element Also wondering how exactly a paid pro version with extra features isn't a violation of the AGPL without those very features being freely available on someone else's public repo

@Tadano @teto @troed @element @skylar it's simple, the entity owns all the copyrights and licenses it under a proprietary license in parallel

@skylar @troed @element

"Use our Community Edition to test the software for free (expires after 14 days)."

@element @teto @troed Why not sell enterprise support instead then? Red Hat and Canonical built multimillion dollar businesses on this, and I'm sure there's a niche Matrix can exploit. Businesses are either still on XMPP or are had at the nads by MS Teams and/or Slack, there's certainly a market for a viable way out of this while retaining support.

Additionally, you mention hardware costs yet continue to build on Synapse which will inherently scale badly despite the fact that Dendrite exists. It already satisfies most of the Matrix spec, scales much better and desperately needs development resources currently allocated to Synapse. Why was there no push for developing that, especially before deciding to announce a "pro" version?
@Tadano @troed @element matrix is already being used by european govs and military. It would work. They are just greedy.
@teto @troed @element No it's not simple greed. A greedy man would leap at the opportunity to gouge government and military for support contracts without compromising on publicly perceived goodwill. This is a combination of greed and stupidity.
@Tadano @teto @troed @element I don't think that's entirely fair- talking about needing funds shouldn't come with that default assumption. It COULD be, but there's a very strong argument for having some form of revenue so that devs can be properly, and more completely, compensated, to continue growing the business.
@Tadano @teto @troed @element I think Enterprise support is a proven business model that would suit something like Matrix especially very well.

@element @troed From my experience large scale deployers of software (also FOSS) always want an enterprise SLA, in case problems pop up they can't handle. Wondering who those large deployment orgs are that feel they can go without enterprise support.

@sunweaver @troed the problem is mainly that system integrators turn up saying they can provide that support and sla by using the foss project, and then try to save costs by not working with us as the upstream - meaning the upstream isn’t funded, undermining the underlying project, and increasing the chances of the delivery going wrong without expert support (and with a weakened upstream).

@sunweaver @troed heise.de/news/Probleme-mit-Ope is one concrete example, which contributed directly to layoffs at Element in 2022 and 2023 as mentioned in matrix.org/blog/2022/12/25/the. Synapse Pro is an attempt at us giving SIs a much more valuable reason to work with us as the upstream rather than trying to cut us out.

heise online · Hessen führt kurzfristig Webex ein: Probleme mit Open-Source-Videokonferenz-ToolBy Christian Wölbert

@element @troed The company mentioned in that Heise post is well-known for scrounging in OSS and not giving back much. Been there, seen that.

@element@mastodon.matrix.org @troed@ioc.exchange Stop developing proprietary software!

If not Ejabberd is free, we will install this instead.

​:gnujihad:​

@element @troed

Who cares if someone profits without giving back when you are supposed to be funded by governaments to power sovereign infrastructures.

@alxlg @troed we care, because if the govt’s money goes to an SI rather than us, we can’t pay people to improve or indeed maintain our software…?

@element @troed

Contact the institutions and explain that they need to support your work in addition to their suppliers? It's not like adopters of an innovative decentralized protocol can't understand how much important it is to support its development.

Or adopt Open Nitrate Model? youtube.com/watch?v=STNomD9GUJY

Going proprietary for the public sector totally defeats the purpose of tech sovereignty and you will have a hard time trying to sell the only viable implementation of a (open) protocol.

@element

You're absolutely correct. The large SIs also don't care if the total cost turns out to be larger since they're not the ones that pays. They only care about their profit.

@landgren has exposed such behaviour in Sweden, as an example. We also have Jens Nylander (don't think he's on Mastodon) who has been documenting public sector waste of money.

This needs to be solved through general awareness at the public sector level - and it is an interesting topic.

@alxlg

@element @troed synapse pro is assuredly based on other open source software, potentially even existing community matrix protocol libraries like ruma

how are you ensuring that you yourself are not freeriding on these community projects? there is not much of a difference between selling server software while not giving back to open source devs and people selling your service without giving you money.

@element What is the motivation to maintain two different codebases (Python and Rust) when the community version is not profitable? Isn't it cheaper to have one version for both paying and non-paying users?

@michal FOSS Synapse is the primary project, and is where all new features and maintenance happens. However, it doesn't fund itself, so the cost of also maintaining a paid Rust port of it is worth it if it lets us keep developing and releasing normal Synapse to everyone as FOSS.

@element @michal what guarantees do we have that foss synapse remains the "primary product"? The recent move away from foss really isnt giving me any confidence that foss synapse will remain usable for the future.

@element It’s ok, not everyone can be good at engineering! The important thing is that you tried and did your best :shiblobheart3:

@element

So... you just reinvented paywalling?

What the actual fuck, how is Matrix an open protocol if its maintainers paywall its features

@laxla @element Well, this is just like KNX and similar stuff. the Protocol is "Open", but the reference implementation and their docs are paywalled.

@JmbFountain @laxla Synapse isn't a reference implementation (and hasn't been for years), and there are no user-facing features here which are paywalled: just massive clustering. Meanwhile the protocol is @matrix.org and is 100% open, just as ever.

@element @JmbFountain @laxla that's not really fair to say. While synapse is "officially" not the reference implementation, it practically is. If synapse chose one way to interpret the spec, other server implementations need to follow that, or run in to incompatibilities with basically all of the matrix ecosystem. You have created a practical monopoly on server implementations with synapse without a feasible migration path to another server.

@laxla Matrix is an open protocol because anyone can, and does, implement it - and Matrix is managed by @matrix.org, not Element. Synapse is just one implementation, and no features are being paywalled here (unless you count huge-footprint elastic scaling as a feature).

@element

Synapse is just one implementation

Isn't it affiliated with matrix.org? That turns it official (even if it's not associated with the .org directly), with an expectation that it'll remain FOSS.

Element and Synpase are the main Matrix client and server; they're the first in Matrix's implementation list. That creates expectations of officiality.

unless you count huge-footprint elastic scaling as a feature

I do, actually. Scaling is a massively important feature for a social network meant to run government websites.

@laxla

> Isn't it [synapse] affiliated with matrix.org?

Not since 2023, when Element had to stop donating Synapse dev to matrix.org given lack of funding for it: element.io/blog/element-to-ado. These days Synapse is just a vendor impl.

> I do, actually. Scaling is a massively important feature for a social network meant to run government websites.

Well, if we had another way to encourage massive Matrix deployments to fund dev, we'd take it.

Element Blog · A new home and license (AGPL) for Synapse and friendsElement has chosen to pursue future development of Synapse, Dendrite and associated server-side projects under the terms of AGPLv3.

@element

Not since 2023

Well, I haven't heard about it. Interesting.

Well, if we had another way to encourage massive Matrix deployments to fund dev, we'd take it.

Well, I see two options.

  • Become non-profit and offer a 'membership' fee that'll provide branding usage rights. I don't know how fitting this model is for you.
  • Offer consulting/installation/pay-for-feature services.

But the paywalling approach is just, why? Like, your corporate/government users have a financial incentive to pay for maintenance anyway, no company is gonna self-install with no warranty whatsoever if they can just not.

This'd be fine if you weren't still the main, semi-official, Matrix quote "vendor".

@laxla

> Become non-profit and offer a 'membership' fee that'll provide branding usage rights.

The Matrix.org Foundation is such a non-profit, selling membership fees (matrix.org/support) - but that didn't generate enough $ at all to fund Synapse dev. Branding rights aren't valuable to commercial deployments, given they typically whitelabel.

@laxla

> Like, your corporate/government users have a financial incentive to pay for maintenance anyway, no company is gonna self-install with no warranty whatsoever if they can just not.

That's what we thought. But what happens is that a big System Integrator turns up to the govt and says "hey, we'll run your Matrix instance for you!", and thinks they can run FOSS and maintain it themselves. Zero $ goes upstream to us, and then you end up with heise.de/news/Probleme-mit-Ope. And it keeps happening.

heise online · Hessen führt kurzfristig Webex ein: Probleme mit Open-Source-Videokonferenz-ToolBy Christian Wölbert

@element alright then. It is, however, a dangerous path you're taking. Lots of companies have been through this and lots have gone full corporate.

But, the question that begs to be asked is: how can these SIs approach the government before you do?

@laxla The SIs are often the only ones qualified to sell to the government; they may already be under contract in an existing general purpose agreement, or they meet the necessary size criteria (turnover, # of employees, insurance thresholds). Plus to successfully deliver a huge deployment typically *does* require a SI to do all the glue legwork. But given a chance, it is just too tempting for the SI to save costs by not paying the upstream. So: you have to give them something to pay for.